Un-Expose Public Control Classes

I have a problem making a custom control. When I create and build the control is in my toolbox which is kind of what I was after:

Public Class PanelBar
    Inherits Panel

    Private _mCusBtn As CustomButton

    Public Sub New()

        InitializeComponent()

        _mCusBtn = New CustomButton()
        AddHandler _mCusBtn.ButtonClicked, AddressOf CustomButtonClicked
        Controls.Add(_mCusBtn)

    Public Sub CustomButtonClicked(ByVal btn As CustomButton, ByVal buttonId As Int32)

        ' Do important stuff here...

    End Sub

End Class

However, when It displays in the toolbox this control also displays:

Public Class CustomButton
    Inherits Button

    Public Property BtnId As Integer
    Public Property BtnColor As Color
    Public Event ButtonClicked(sender As CustomButton, buttonId As Int32)

    Public Sub New()

        ' Set new property values

    End Sub

    Private Sub CustomButtonClicked(sender As Object, e As EventArgs) Handles Me.Click
        RaiseEvent ButtonClicked(Me, BtnId)
    End Sub

End Class

So I have tried setting the CustomButton class to friend to limit the outside access because I do not want this as a control in my toolbox and I get this error: 'btn' cannot expose type 'CustomButton' outside the project through class 'PanelBar'. on the CustomButtonClicked event of the first class.

I'm not sure if this makes sense to want to limit the scope of the controls that help make up my custom control since it is having to access the events on them. To be honest I do not recall one time that I have come across a custom control that doesn't list the controls that make it up, so I am not entirely certain this is going to be possible...but I would really appreciate any advice I can get.