Facebook's Messenger Platform lays the groundwork for more. Source: Facebook.

Remember when Facebook (NASDAQ:FB) tried to make its own smartphone operating system? It was a complete flop, as most users aren't willing to give Facebook total control over their smartphone regardless of how much they use the social network's apps. Apple (NASDAQ:AAPL) and Alphabet's (NASDAQ:GOOG) (NASDAQ:GOOGL) Google still control nearly the entire mobile operating system market.

But Facebook hasn't given up on its ambitions to be more than a social network. That ambition is clearly seen through the changes to Messenger over the past year or so. The company is bringing on businesses to interact with customers, developing its own virtual assistant, and most notably it built the foundations of an app store on top of the platform. As Facebook continues to expand the role of Messenger, it could run into trouble with Google and Apple.

The Messenger app store
In March at its F8 developers conference, Facebook announced Messenger Platform, which allows developers to create apps that integrate more closely with Messenger, enabling users to send pictures or videos to one another through Facebook's app. Developers benefit from the free publicity, as Messenger features a select number of apps within Messenger and includes an install button when a user sends a picture to a friend who doesn't have the app installed yet.

But Messenger Platform still relies on the underlying app stores from Google and Apple to make it work. Ultimately, Facebook could get the most utility from its own app store built on top of Messenger. Users could switch devices and platforms without losing data or paid app installs.

Pretty much everything is in place for Messenger to build a full-fledged app store. Facebook already started collecting payment details in Messenger to enable peer-to-peer payments. The company also has an app payments platform for Web apps that were built on Facebook. (That business still brought in over $200 million in the second quarter, by the way.) And it already has partnerships with several developers for apps.

The only problems that stand in its way are Google and Apple.

It's easy to get around Google, but Apple will fight back
Neither Google nor Apple allows third-party app stores in their respective mobile app stores. Android users are capable of sideloading apps from a website, but that option isn't available to iPhone owners.

The real problem is that, ideally, Messenger would become an app store, much like Weixin (WeChat) in China. Facebook's head of messaging, David Marcus, has noted that he looks to WeChat as an example of how to monetize Messenger. The app store within an app model has worked extremely well for Weixin in China, where smartphone owners are used to third-party app stores. However, integrating a similar experience in Messenger may cause both Google and Apple to ban Messenger from their app stores.

And that could be an issue for Facebook, since it's still growing the number of users on Messenger -- 700 million at last count. As such, Facebook will probably keep any entry into third-party app stores as a separate application from Messenger, resulting in a less than ideal experience.

Just like Home -- Facebook's failed OS -- a Facebook app store would have to start on Android. Facebook can easily direct users from its flagship app or Messenger to install the app store on the Web.

If Facebook gets good traction with an app store, however, it will want to expand to iOS. That's when the question becomes, "Is Facebook worth more to Apple than Apple is worth to Facebook?" It's hard to imagine that the iPhone would sell as well without support for Facebook, Instagram, Messenger, and WhatsApp, but it'll be almost impossible for Facebook to perform well without the high-value Apple customers who use iPhones. At this point, Apple seems to hold more leverage over Facebook, with its 500 million iPhones in use.

But why an app store, anyway?
As I mentioned, an app store opens up a new opportunity to monetize Messenger. Beyond that, however, Facebook has already shown an ability to recommend new apps better than both Google and Apple, which would make it attractive for developers looking for targeted exposure. Additionally, an app store is the perfect place to display app-install ads, something Google is just finally getting around to doing in Google Play. Facebook is the undisputed leader in attracting developers to its app-install ads (probably because of its targeting capabilities) and could take advantage of the huge demand for the ad units on its network with an additional place to display those ads.

Rest assured Facebook will take its time to further develop an app store, probably on top of the Messenger Platform. At this point, it doesn't look as if it will be able to get Apple on board, but if it takes off on Android, it could be worth it to take on Apple.

Adam Levy owns shares of Apple. The Motley Fool owns shares of and recommends Alphabet (A and C shares), Apple, and Facebook. Try any of our Foolish newsletter services free for 30 days. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. The Motley Fool has a disclosure policy.